Skip to content

Support chinese paths, token login, verbose debug, multithreaded download & more #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

Support chinese paths, token login, verbose debug, multithreaded download & more #102

wants to merge 10 commits into from

Conversation

425zekunwang
Copy link
Collaborator

What's Changed

This PR includes a series of enhancements for the CSGHUB CLI tool, focusing on usability, reliability, and performance during upload/download operations.

Key Improvements:

  1. Chinese Path Handling

    • Fixed an issue where uploading repositories with Chinese characters in the path would fail.
    • Automatically checks if git config --global core.quotepath false is set (which allows proper display of non-ASCII paths).
    • Detects whether Git has a user.email and user.name configured, and prompts the user to configure them if missing before upload.

  2. Token & Login Management

    • Added a new --login command-line argument.
    • Supports saving the user token either via --token <your_token> or by reading from the environment variable.
    • Tokens are securely stored in the user’s home directory.

  3. Download Structure Improvement

    • When downloading a repository using csghub-cli, it now automatically creates a directory structure based on the repo_id under the current working directory.

  4. Verbose Debug Mode

    • Introduced a --verbose flag to help users and developers see detailed execution logs, making debugging easier.

  5. Optimized Upload Logic

    • If there are no changes in the repository, the upload will be skipped, and a message nothing to commit will be shown.
    • Supports incremental uploads (only changed content is uploaded).

  6. Multithreaded Downloading

    • Implemented multi-threaded downloading for repository files, significantly improving download speed and efficiency.

  7. SDK Installation & Configuration

    • Improved the installation and configuration process for csghub-sdk, ensuring smoother setup and dependency resolution.

  8. Download Progress Feedback

    • Added a callback mechanism in snapshotdown to provide real-time download progress information (e.g., percentage, speed).

  9. Chinese Repository Name Check

    • Checks if the repository name contains Chinese characters during upload and warns the user accordingly.

Notes for Reviewers

  • This PR touches CLI behavior, token/auth management, download logic, upload stability, and user experience.
  • No breaking changes introduced.
  • All new features are behind optional flags or improvements that preserve backward compatibility.

425zekunwang and others added 7 commits August 6, 2025 18:05
@425zekunwang
feat: add build script and update version handling
7eba5f1 
- Introduced a new build.py script to manage version updates from pyproject.toml.
- Updated __init__.py to reflect the new versioning approach.
- Enhanced README files for better clarity and formatting.
- Minor adjustments in example scripts for consistency.
@425zekunwang
refactor: update documentation and logging for clarity
910ef66 
- Translated comments and docstrings from Chinese to English for better accessibility.
- Improved logging messages in various modules to enhance clarity and consistency.
- Updated version in __init__.py to 0.7.5 to reflect recent changes.
- Minor adjustments in example scripts to align with updated logging and documentation.
@HaiHui886
feat(docs): update README and add examples
362a1e2 
- Simplified introduction text
- Improved formatting for installation instructions
- Clarified download location for files
- Added examples directory with usage scripts
- Updated Chinese README for consistency
@HaiHui886
feat(datetime): patch datetime parsing method
5efb886 
- Add custom datetime parser for RFC 3339
- Mock patch to replace huggingface_hub method
- Ensure compatibility with older versions
@HaiHui886
fix: update default version and clean up code
5eeda6a 
- change default version to 0.1.0
- remove allow_patterns from snapshot_download calls
@HaiHui886
fix(download_dataset): remove allow_patterns argument
f3fda2d 
- Removed allow_patterns from snapshot_download call
@HaiHui886
fix(dependencies): remove unused packages
721f317 
Removed tqdm and requests from dependencies.
@HaiHui886 HaiHui886 changed the title feat: support chinese paths, token login, verbose debug, multithreaded download & more Support chinese paths, token login, verbose debug, multithreaded download & more Aug 19, 2025
@HaiHui886
feat(download): add parallel download support
0481816 
- Added use_parallel option for downloads
- Introduced max_workers parameter
- Refactored snapshot_download for threading
- Created single and multi-thread download methods
@HaiHui886
fix(snapshot_download): fix parallel download logic
62322f1 
- Restore call to snapshot_download_with_single_thread
- Pass max_workers to snapshot_download_with_multi_thread
@HaiHui886
feat(logging): replace print statements with logger
81ae2bd 
- Use logger for token retrieval messages
- Replace print with logger in login/logout
- Update download/upload error handling to use logger
- Change verbose flag to enable_parallel in download
- Update snapshot_download to use logger
- Replace print statements with logger in utils
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@425zekunwang @HaiHui886